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ABSTEACT 

A nationw.\de, longitudinal study investigated changes 
IXL Student achievement in grades 1-12 associated with intensive 
educational innovation. Standardized reading, language, and 
arithmetic test scores; student and teacher background and 
a.ttitudinal data; and educational characteristics underlying school 
programs were obtained over a three-year period. Findings showed a 
lack of either substantially or consistently positive associations 
between achievement and program-level emphasis on innovation, 
Eelative to the sample of schools participating in the project, 
students enrolled in programs with a more moderate emphasis on 
innovation and individualization showed, the greatest improvement, 
JLounger students (those in grades 2 and 3) who achieved substantially 
more than expected during two consecutive years were exposed to a 
notably greater amount of class time per day on language arts during 
the second grade than were the consistent underachievers. The results 
suggest that educational innovation per se will not necessarily 
produce dramatic effects on student achievement. Educational quality 
is not synonymous with innovation, (Author) 
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THE UNFULFILLED PROMISE OF EDUCATIONAL INNOVATION 



Albert B. Chalupsky 
Gary J . Coles 

American Institutes for Research 
Palo Alto, California 

Backjuround 

During the ]ate 1960s and early 1970s hopes were very high that 
intensive, innovative educational programs would demonstrate dramatic 
impact on student achievement. While these hopes have moderated in recent 
years, there is stxll the feeling among many individuals, government offi- 
cials as well as the general public, tliat if schools undergo major change 
they will somehow do much better. As a result, many efforts continue to be 
sponsored at the federal and state levels to encourage such change, either 
through the development of -lew programs or\ through increased emphasis on 
the dissemination/diffusion of innovallons ^ educational "change agents." 
The ^'ii^olesale adoption of educational innovations, however, is an expensive 
process, and Lherc is no guarantee that tne added costs wi J 1 neressanjy he 
justified by improved program effectiveness. 

Educators and noneducators alike have shown a growing awareness of 
the lack of-- and need for— evidence as to whether or not innovative edu^^-"' 
tional practices are indeed better than the more traditional approaches. 
In response to this need, th^. U. S Office of Education in 1969 awarded a 
contract to the American Institutes for Research to develop a design for a 
study of the overall effectiveness of highly intensive, innovative educa- 
tional practices on students in grades 1 through 12. The initial design 
was developed, implemented on a limited basis, and modified during the 
1969-70 school year. Full implementation began during the 1970-71 school 

Project Objectives 

The specific objectives of Project LONGSTEP (tlie Longitudinal Study of 
Educational Practices) were as follows: 

• to design a system for studying the educational environments of 

students in terms of meaningful dimensions which underlie various 

educational approaches or "treatments"; 



• to establish a . large-scale data base consistinj; of treatment 
characteristics, student characteristics, and educational outcomes 
for a select sample of educational programs involving intensive 
and highly innovarive educational practices; 

• to determine, as comprehensively as possible over a three-year 
period of time, the impact of intensive innovation upon student 
performance on standardized achievement tests and on measures of 
educationally relevant attitudes; and 

• to attempt: to identify the dimensions of educational components 
(present in a select sample of highly intensive, innovative pro- 
grams) that exhibit the greates't impact on studen;- outcomes. 

For purposes of this study, "intensive innovation^' meant the implementation 
of a new progran encompassing a significant proportion of students, entail- 
ing a major alteration of school procedures, and involving a high invest- 
ment of resources. 

Study Apprca c h 

The general emphasis of Project LONGSTEP was on the identification of 
changes in student achievement that occur as a result of intensive educa- 
tion.-! innovation. Highlights of the methods used to identify and docu- 
ment innoyations and to assess student achievement are discussed next. 
Sejection of Schools and Students 

During the early stages of the study, a nationwide search was -jnrler- 
taken to locate educational programs incorporating intensive, innovative 
practices. Guidelines for program selection emphasized program scope and 
intensity, instructional content, anticipated program" continuation, and. 
willingness to cooperate in a multiyuar study.' 

Through a combination of extensive literature sl .ch, interviews with 
program staff, outside consultant review, and site visiting, schools 
located in lA school districts were selected and agreed to participate. 
Of these 14 districts, 13 participated throughout the entire implementa- 
tion phase and provided the data on which this report is based. Schools 
and districts were located in nine states: California, Florida, Kentucky, 
Minnesota, Pennsylvania, Texas, Utah, Washington and Wisconsin. Altogether, 
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seme 30,000 studonts, 80 schools, and 1,500 tcichers parr.icipated in the 
study during its three years of implementation. 

At the time of this study, most of the par (: icipatinp, educational 
programs were supported largely by local funding. However, many had their 
roots in educational research and development activities least par- 
tially funded by the federal government. 

Samp 1 e De c r ip t i c n 

The 13 districts participating in Project LONGSTEP provided variability 
along various dimensions, as illustrated by a number of school and community 
characteristics noted during the 1970-71 school year. The communities 
served by the school districts ranged from 2,500 to over 600,000 in popula- 
tion and varied from rural to urban-metropolitan in setting. Their diver- 
sity in socioeconomic level is notable, as evidenced by the percentage of 
students in the participating schools receiving free or reduced-price 
lunches. This number ranged from less Uhtfn 1 percent to nearly 30 percent. 
The report-pd instructional cost per pupil in the participating school 
districts varied from a low of $54( to a high of $1,050. Another indicator 
of the diversity in the sample was the percentage of minority group students 
in the participating schools; this ranged from less than 1 percent to over 
30 percent. 

The educational innovations encor.ipassed by Project LONGSTEP were those 
that V7ere of particular concern during the late 1960s and early 1970s — 
most are still of concern today. These innovations included team teaching, 
multimedia emphasis, unique school design, use of paraprofessional s , 
variations in scheduling, and teacher-developed materials, as well as 
independent study, student selection of materials; and a number of other 
practices typically associated with individualized instruction. An over- 
view of the major educational features present in our sample at.tlie time 
programs were selected, as well as the distribution of such features 
across districts, is presented in T.lble 1. 

Caution sliould be exercised in drawing conclusions froLi this table, 
however, since the same label may have encompassed significantly different 
activity elements from one school district to another. By the same token, 
different labels were often applied to vory similar activities. Further, 
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TABLE 1 

Overview of Major Educational Features Occurring in One or More 
Schools of the Districts Participating in Project LONGSTEP 



Major Educational Features 


School Districts 


^ 

A. 


2 


3 




5 


6 


7 


8 


9 


loln 




13 


Small-group activities 


X 


x 


X 


X 








X 


X 


X 




X 


X 


Team teaching 


X 


X 


X 


X 


X 


X 


X 


X 


X 










Multimedia emphasi s 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 




X 




Ungraded curriculum 




X 


X 


X 


X 




X 


X 


X 








! 


Large-group instruction 




X 


X 










X 


X 


■X 


X 


X 


X 


Independent study 


X 


X 


X 


X 


X 






X 




X 




X 


X 


Unique school design 




X 


X 








x 


X 












Differentiated staffing 


X 


X 








X 
















Volunteer aides 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 








"^PI reading/math 




X 






X 


X 






X 










Project PLAN 














X 




X 










Individualized ins true t ion 
(other than PLi\N or IVl) 


X 


X 




X 




X 


X 


X 


X 


X 




X 


X 


Student selection nf m^1^r»r^n^c 


X 


X 


X 






X 




X 


X 


X 








Modular scheduling 


X 


X 


X 


X 








X 












Teacher sensitivity training 












X 
















Intensive guidance and 
counseling 












X 
















Home visitation 












X 














X 


Traditional/ conventional 
classrooms 


X 




X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Teacher-developed curriculum 
materials 


X 


X 


X 


X 




X 




X 


X 


X 


X 


X 


X 
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it should' be stress.cd that the educational activities "listed in tl:e table 
were often given these designations by the school systems. They did not 
necessarily reflect thc^ underlying educationpl processes that were tlie 
primary cbncern of this study ♦ Nevertheless, Table 1 does provide a 
gross indication of the wide variation that existed in the Project LONGSTEP 
sample. 

Data Collection Instruments and Schedule 

The data colio'^tion instruments used in this study provided informa- 
tion on student cognitive performance, student characteristics, educational 
experiences, and teacher characteristics. Student cognii:ive achievomont 
was measured by either of two commercially developed instruments, the 
Comprehensive Tests of Basic Skills (CTBS) and the California Short Fonn 
Test of Mental toturity (CTMM) , 1963 revision. The CTBS was used for' 
spring testing in grades 2-12 and the CTMM for testing in grado 1. Student 
and teacher background characteristics and attitudes were assessed by 
questionnaires developed specifically for Project LONGSTEP. 

In order to investigate the relationships between educational prac- 
tices and educational outcomes in a very diverse group of schools and pro- 
gram.^:, a system was needed for describing and quantifying the educational 

experiences that each student had during a school year. It becaine evident 
early in -the development of this system that, although there are a limited 
number of labels used by school district.s to describe their educational 
activities, in practice there is a wide disparity amor.g educational 
approaches sharing the same descriptive label. For example, ^'modular 
scheduling" is a commonly-used term for what are in fact various . conf i dura- 
tions of class schedules. At the same time, programs bearing different 
labels often turn out to be very much alike. Thus, one district's program 
of '^individualized instruction'' could be much the same as anotlier district's 
"nongraded" program. Therefore, an Educational Experience Analysis Guide 
was developed so that complex educational experiences could be described 
and quantified with resp^-ct to specific observable characteristics (rather 
than on the basis of variously defined local labels;, AIR staff used this 
guide to document the basic educational attributes of the scliool programs 
in. which participating students were enrolled. Information on .educational 
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experiences was gathered from interviews with principals and Leachors, from 
classroom observations, and from existing scliool documentation. 

The guide made it possible to locate the educational experiences of 
participating students on a continuum ranj;inj; from traditional to innova- 
tive. Use of the guide also ensured that all students in a school who 
were receiving essentially tho. same basic educational experiences would be 
identified as belonging to the same '^educational experience group" and be 
distinguished from students receiving different educational experiences, 
even though both groups of students may have been participating in the 
same school '^program.*' 

During the 1970-71 school year, this' documentation methodology led to 
the identification of I/4I educational experience groups; during the 1971-72 
and 1972-73 school years, respectively, 167 and 228 groups were isolated. 
These groups represented all the different kinds of basic educational expe- 
riences provided by the schools for students participating in Project- 
LOFGSTEP, An illustration of the diversity of our sample is shown. by the 
characteristics of the five school districts that contai^ned language arts 
groups for the third grade during the 1972-73 school year. Figure 1 
shows the average scores for these five districts on ten key educational 
indices (derived from items on the Educational Experience Analysis Guide) , 
plus student socioeconomic status, and the number of minutes per day of 
class time spent on language arts activities. 

One final data collection guide, a classroom observation instrument, 
was developed to document classroom characteristics such as physical envi- 
ronmeiit • study arra.igements, and access to resources, as well as teacher 
and student activities such as degree of grouping, focus of activities, 
and use of materials. 

In order to collect all the necessary data on students, teacliers and 
educational experiences, AIR staff member? typically visited each site 

■> 

three tinies during a school year. Site visits generally ranged from five 
to eight: days in length. Docum^enta tion of educational experiences was 
usually accomplished during the initial visit each year. 
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Variable 



Conventional 
or Low 



Mid-Point 



Innovative 
or High 



Socioeconomic 
Status 

Ut-llization of 
Ob jectives 

Indi V idualizat ion in 
Decision Makinp 

Teacher or Locally 
Developed Materials 

Individualization of 
Instructional Pace 



Scheduling 
Charnc t erist ics 



Use of Performance 
Agreements 



Classroom fi^roup 
(^rgani zai ion 



Teaching Unit 
Compos i t ion 

Completeness of 
Instructional Package 

Utilization of 
Student Evaluation 

Number of 
Minutes per Day 



Site No. 'X 0 



A**»^"^cC r'^^'^ 




Figure 1, Profiles for language arts groups for the five districts/ 
sites containing third grade students during the 1972-73 
school year. 
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Ov'erviow of the Data Base 

Project LONGSTEP represents a pioneering effort in educational evalua- 
tion. It is perhaps the first large-scale, longitudinal study to associate 
the educational attributes nnrjprTyin£ srhool proornms and the characteris- 
tics of teachers with only those individual students who were directly 
exposed to their influence. The vast amount of data collected is also 
notable — other studies may have gathered information on more students, but 
for sheer quantity and complexity of data, this study is unique. When alJ 
the tests and questionnair'ijs administered to students and teachers and all 
the Educational Experience Analysis Guide data collected on language arts, 
mathematics, social studies and science treatments over a three-year 
period are considered, some 3,500 individual items of data could be 
associated wir.h a single student. When the scales or indices derived from 
all instruments are also considered, the total number of variables could 
increase to approximately 4,000 per student. 

Because of the potential of such a large and detailed data base, a 
scries of generalized data files and cuueL-uuks wei'-e produced for the 
U.S. Office of Education so that? these data can be made available to 
other researchers. ..^ 

Analysis Approach 

\>/liile Project LONGSTEP involved students in grades 1-12, the analyses 
reported here involved only three groups of students: those who started 
out as first graders in 1970-71, those who started out as fourth, graders in 
1970-71, and those who started out as sixth graders in 1970-71. Tliese 
grade levels were selected for initial analysis primarily because their 
data would permit j the examination of educational growth occurring during 
the primary years (grades 1 to 3) , the elementary years (grades A to 6) , 



(grades 6 to 8) . A detailed presentation of the methods and findings is 
contained in Volume I of the Project LONGSTEP final report (Coles, G. J. 
Chalupsky, A. B., Everett, B. E. , Shaycoft, M. F., Rodabaugh, B. J., and 
Danof f , M. N. , 1976) . • 

10' 
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The measures that were related to posttest achievement included each 
student's socioeconomic status (SES) and pretest score; the educational 
attributes to which each student was exposed, including the overall amount 
of innovation and individualization in the school program; the amount of 
ciass -time alloeaLed by that program to language arts or mathematics in- 
struction; and the qualifications of each student's teachers. Amount of 
innovation was measured by an index called Level of Innovation. This 
scale was equal to the sum of the following scales derived from the 
Educational Experience Analysis Guide: Individualization in Decision Making, 
Individualization of Instructional Pace, Use of Performance Amercements, 
Utilization of Student Evaluation, Utilization of Objectives, Teacher or 
Locally Developed Matdrials, Scheduling Characteristics, Classroom Group 
Organization, Teaching Unit Composition, and Completeness of Instructional 
Package. Our measure of program emphasis on individualized instruction. 
Degree of Individualization, was defined as the sum of the first four 
of these scales . 

Although Project LONGSTEP collected data for three consecutive school 
years, two pretest /posttest analyses (Year 1 verfi.us Year 2 and Year 2 versus 
Year 3) were viewed as the most powerful approacll for the initial analysis 
of these data. Because of the multilevel nature of the study's data base, 
project staff concluded \ that overly complex analytic models should not be 
utilized until the data and the relationships among the basic variables 
were more clearly understood. 

In order to determine whether intensive innovation was related to 
substantial gains in achievement in Project' LONGSTEP 's diverse sample of 
schools, the study assumed that if innovation does have a dramatic impact 
on achievement, then: 

• variation with respect to intensity of innovation, even within 
a sample of generally innovative schools,, should be positively 
correlated with achievement; O 

• such impact should be a general trend across different classrooms-, 
schools and school districts and should be consistently present in 
consecutive school years; 
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• the achievement gains demonstrated by. Project LONGSTEP's generally 
innovative sample of schools should noticeably exceed the gains 
shown by national norms; and 

• such impact should be large enough to be demonstrated by a number 

of different methods, including those involving the kinc^s of statis- 
tical adjustments normally required in analyzing nonexper imental 
data. 

Because Project LOInGSTEP collected data from students in ongoing 
school programs, it was deemed essential that the study's key hypotheses be 

' evaluated from a number of slightly different methodological perspectives 
so as to minimize the possibility that findings would be highly method- 
.<^r assumpt ion-deperdent . Conclusions could then be based on con.verging 

' lines of evidence. For this reason, the analyses were, by design, as iaten- 
'sive as possible within project time and cost constraints. 

Findings in Review 

So that they might be summarized in a concise and meaningful uay, the 
Project LONGSTEP findings have been organized into four sections. Each 
section is headed by the overall, basic question to which the findings 
pertain. 

Were Students of Different So cioeconomic Status and Pretest Levels Exposed 
to Different Kinds of Treatments? 

\ In a number of grades, lower SES and lover pretest students were mem- 
bers of, schools and instructional programs that spent more time on language 
arts and mathematics activities. There a'^lso was a very slight tendency 
across all analyses for lower SES students to be exposed to less innovative 
and* less individualized programs of instruction. Even the most notable of 
these trends, however, was* exti^emely small relative to the magnitude of the 
differences in socioeconomic status among the groups studied. The general, 
finding that student SES and pretest background were not both highly and sys- 
tematically related to the characteristics of educational approaches means 
that it is not likely that such factors biased (in a constant manner across 
all analyses) the overall results of Project LONGSTEP concerning the impact 
of intensity of innovation. 

10 
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To What Extent Is Achievement Related to. Pretest and Socioeconomic Status? 



Both initial achievement level (i.e., pretest) and student socioeconomic 
status (SES) are usually substantially related to posttest achievement in 
school effects studies. The results of Project LONGSTEP are no exception. 
It is notable, however, that: \- 

• pretest was a vastly better predictor of posttest achievement than " 
was SES; 

• the predictive value of SES was reduced almost to zero after the 
predictive) overlap with pretest was taken into account; and, 

• the relationship between pretest and posttest was still very large 
even after student differences with respect to SES and after group 
differences with respect to ' mean pretest level were taken into 
accoun t . 

While not the major purpose of the study, these findings Jiavc implications 
for both educators and researchers. Based on n diverse sample of schools, 
such findinps suggest that when pretest is known^ it is not necessarv to 
use socioeconomic status as a predictor of individual student posttest 
achievement. They also imply that SES cannot be expected to serve as a 
reasonably accurate substitute measure of initial achievement (i.e., 
pretest) level . 

Did Inte r, s ive Ed ucational Innovation Result in Substantial: Growth i n 
Ac I li ev er.jjj n_t ?_ 

Project LONGSTEP findings with respect to the impact of educational 
innovation can be organized into two general categories. Tlie first group 
of findings evaluated the overall amount of educational growth demonstrated 
by each of the samples analyzed and compared this growth with tliat of tlie 
CTBS norm sample. The second group of findings related variation in in- 
tensity of innovation to variation in achievement growth. 

G rowth in achievement and norm comparisons . Comparisons among average 
pretest performance, avetage posttest performance, and national norms 
showed that : 

• the greatest gains in reading, language and arithmet ic achievement 
occurred in the earlier grades; 

IJ 
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• the magnitude of yearly gains in achievement tended to decrease 
with each higher grade; 

e * for all grades, acliievernent gains seemed to be sllglitly larger in 
arithmetic Mian in language arts (reading and language); 

• although gains were not dramatic, achievement growth during tliird 
grade was somewhat greater than that in the norm sample; and 

• Project LONGSTEP students, on the average, did not do conspicuously 
better on posttcst (relative to national norms) than they did on 
pretest . 

In summary , the expectation that substantial yearly gains in student 
achievement wouJ.d occur for a sample of intensive, innovative educitional 
programs is not supported by these findings. 

Grov/th in achievement and innovative emphasis . The most acceptable 
approach to analyzing data obtained by nronexperiraental/associational 
studies (like Project LONGSTEP) is to use a number of different analysis 
methods r^^'^^^ look for consistent results across methods- Therefore, 
Project LONGSTKP analyzed: 

• overall differences in acliievt^'ment growth among educational 
approaches; 

• an educational growth model in which achievement was related to 
innovative emphasis, number of minutes of instruction per day, 
pretest, socioeconomic status (SES) , and teaching, qualifications; 

• a posttcst achievement score statistically adjusted for pretest 
and SES differences; 

• p.roups of students -i^itli similar educational experiences vlio , on 
the average, performed either much better or much worse than was 
Cixpected from their pretest and SES; and 

• consistently overachieving and underachieving students. 

The major findings based on these methods are reported next with respect to 
reading and arithmetic achievement. No separate discussion is presented 
for language achievement since those findings closely paralleled those for 
reading. A later section- contains a more general summary of project finding', 
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Rcadinc achievement . The analyses of reading achievement indicated 

that: 

• meaningful differences in educational growth did exist among 
different educational experience groups, especially those 
involving students in the early elementary grades; 

• differences among educational experience groups decreased with 
each higher grade; 

• in spite of group differences in acln evemcn t , no consistent over- 
all relationship exifsted between innovative intensity (as indexed 
by the 'study's measure of general innovativeness , Level of Innova- 
tion) and posttest reading performance, or between individualiza- 
tion emphasis (as indexed by Degree of Individualization) and post- 
test reading performance; 

• in the only series of analyses in which growth in reading acln'eve- 
ment was related to overall innovation and individualization to a 
substantial degree (during the third grade), the impact of Level 
of Innovation (or Degree of Individualization) was negative — that 
is, the greatest educational growth occurred in programs with a 
more moderate emphasis on innovation; 

9 the study's measure of teacher experience , Teaching Qualifications, 
was not related to posttest performance to any meaningful degree ii; 
our analyses; and 

• the analysis of unusually effective reading groups provided no 
clear evidence that .Level of Innovation or Degree of Individuali- 
zation was consistently related to achievement. Tliis was also 
true of the analysis of students wlio demonstrated significant 
overachievement oi underachicvement during two consecutive school 
years. In fact, the consistent overachievers actually tended to 
be members of programs with a lower Level of Innovation or De^;ree 
of Individualization . 

An additional finding regarding the study's measure of amount of 
schooling, number of minutes per day of class time on language arts, is 
noteworthy. Younger students (i.e., tliose in grades 2 and 3) who c^xhihitod 
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unusually large gains in reading acliicvemonc during two consecutive school 
years were exposed to much more language arts instruction as second graders 
than were students with a notable lack of growth (i.e., 113 minutes pur day 
compared with 85 minutes). The consistent overachievers , as third graders, 
were exposed to about 20 minutes of instruction per day less than during the 
pre-ious school year, and yet they again demonstrated la.ger than expected 
gains in adiievement. These results sugges t . tliat increased exposure to 
language arts during the second grade may significantly improve the chances 
of some students to demonstrate substantia] gains in reading achievement, 
even in letter grades , 

Ar i t limc t ic /Ma t he ma tics a c h i e vemen t . Meaningful differences among 
educational experience groups existed with respect to arithmetic postrest 
achievement, especially at the li)wer grade levels. In fact, differences 
among these groups decreased witli each higher grade. Other analyses sliowed, 
that : 

• Level of Innovation or Degree of Individualization was negati>/ely, 
rather than positively, related to growth in aritlimetic achieve- 
ment. This appeared to be the most notable result in the samples 
as a whole. Therefore, the hypothesis that greater overall empha- 
sis on innovation or individualization is positively and consis- 
tently related to student achievement in arithmc*tic was clearly 
not supported. 

• Neither Level of Innovation nor Degree of Individualization demon- 
strated a dramatic or consistently p osi t ive effect on student 
arithmetic ^ichievement in (1) i u , lalyses of unusuaJ ly effective 
educational experience groups and (2) the analyses of students who 
demonstrated significant overachievement or undorachi evemdnt for 
two consecutive school years. In fact, relative to under/ichieving 
students, the students who were identified as overachievers, on 
the average, were members of programs with a more moderate empha- 
sis on innovation or individualization. ' 
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O vGrview of achievement and innovative emphasis . In summary, no 
evidence could be found that either of the major treatment variables-- 
Level of Innovation and Degree of Ind ividual izat ion— was substantially 
and positively correlated with posttest performance. Further, preliminary 
analyses showed f:hat, on the average, these findinp,s applied equally well 
to students at different SES or pretest levels. 

Meaningful differences in the teacher qualifications of groups per- 
forming at a higher or lower level than predicted were infrequent. Wlien- 
ever they did occur, however, better teaching qualifications (as indicated 
by such items as education, teaching experience, certification and type of 
appointment) were associated with better student performance. And finally, 
younger students (i.e., those in grades 2 and 3) who achieved subs tant ia 1 J y 
more than expected during two consecutive years " were expose^! to a notably 
greater amount of class time per day on language arts during', the second 
gradi' than were the consistent underachxever.s . Even though they spent 
considerably less class time per day on language arts in grade 3, the 
overachievers agarn demonstrated dramatic gains. 

Can Schools Have a Substantial Impact on Educational Achievement? 

The findings show that substantial , educational growth was not uniquely 
associated with innovative emphasis. Even small amounts of growth were not 
positively and consistently related to our measures of innovation. Level of 
Innovation and Degree of Individualization. It should not be concluded, 
however, t!iat substantial growth was absent in the schools and students 
participating in Project LONGSTEP. On the contrary, as discussed previously, 
a number of educational experience groups were identified whose students, 
on the average, did much better than was predicted from their pretest and 
SES level. For example, as indicated in Figure 2, the average achievement 
gains of sixth grade students in the most effective reading treaf.ment groups 
differed dramatically from those exhibited by students in the groups that 
did much worse than was predicted from their pretest score and their socio- 
economic status. The students in the poorer performing reading groups 
shown in Figure 2 had average pretest scores ranging between the 56th and 
Z4th percentiles; their average posttest scores dropped slightly, ranging 
between the 41st and 66th percentiles. In contrast, the sixth grade 
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Figure 2. Educational experience groups where students achieved 
more or less than was expected from their pretest and 
SES level (Spring 1973 Reading Analysis) . 
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students in the high puriorming reading groups, who i)aG ;u'c-rnge pretesi 
scores ranging between the -8th and 69th percentiles, improved noticeably, 
with average posttest scores ranging between tlie 63rd and 79tli percentiles. 

Other examples of dramatic gains in acliievement were shown in the 
analyiies identifying individual students who, for two con>;ecut i ve VL-ars, 
demonstrated posttest achievement differing greatly fmw that expected on 
the basis of their- pretest and SES scores. 

The analyses ol unusually effective groups and thc^ analyses cr jtu- 
dcnts with unusual growth patterns both provide evidence tliat\ large gains 
in cognitive achievement were occurring in Project LONGSTKP. Thus, even 
though the overall project findings showed that dramatic school eifects 
were not associated with intensity of innovation, the findings noted liere 
suggest that substantial achievement gains, over and above those expected 
on the basis' of pretest and SES, do occur in public schools. It should 
also be remembered that the overall analyses showed that small but meaning- 
ful posttest differences existed among our educational experience groups, 
although they were not uniquely associ^»-^H witli the project*? me^snrf^R of 
innovative intensity. 

These findings certainly are not surprising to teachers, principals, 
parents or even researchers who have personally witnessed impressive growth 
in cognitive skills. They are, however, somewhat more "newsworthy" in terms 
of educational evaluation results. Taken together, then,, these findings 
must. lead us to conclude that: 

• different educational approaches do produce meaningful and 
important differences in achiG*v(^;men t (especially in the earlv 
elementary grades); and 

• although substantial gains in achievement were not demonstrated 
by a majority of Project LONGSTEP programs/schools and students, 
unquestionably dramatic improvements in reading, language and 
arithmetic skills wf^re found in some schools. 

Future research and evaluatioji efforts should be directed at identifying 
the determinants of such significant educational gains. 
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Overa33 Conclusions and Implications 

The single most important and well-documented finding', of Project LONGSTEP 
was the lack of either substantial or consistent association between student 
achievement and overall Level of Innovation across grades . In fact, rela- 
tive to the sample of schools participating in the project, students enrolled 
in programs with' a more moderate emphasis on innovation and ndivi duali za~ 
tion sliowed the greatest, improvement. 

From the standpoint of time spent per day on language arts activities, 
it appeared that consistently overachieving second and tliird graders in 
Project LONGSTEP typically were members of programs that allocated more 
time to such ins true ti on - in the second grade. These findings suggest tliat 
increasing the amojnt of class time per day, especially in the first and 
second grades, may be a worthwhile strategy for improving student performance 
in language arts. 

Our various analyses also suggest that the impact of th^'i educational 
practices studied was inversely related to, grade level. Specifically, it 
appeared t!iat differences among educationa.' experience groups, inde])en'^'ent 
of the effects of student SES level, initial achievement level, and Teaclier 
Age, SES liackground and Teaching Qualifications, were greatest for second 
and third graders, less for fifth and sixth graders, and almost nonexistent 
for seventh and eightii graders. These findings tend to- support the view 
that the greatest potential payoff from the investment of future educa- 
tional research and instructional development funds may cqme from increased 
attention to improving education in the early elementary grades. It is 
suggested that further effort be expended to gain a better understanding of 
the educational practices that show the greatest benefit at these early 
grades. The results of sucli research could then serve as the basis for 
designing educational strategies that could have a relatively greater 
overall impact on American education. 

It is less obvious why one particular aspect of much recent innova- 
tion, that is, increased individualization of education, lias not been sliown 
to be clearly and consistently effective in Project- LONG.STEP. Some pos- 
sible explanations for this finrling will be discussed, but it should be 
emphasized that these are purely speculative. One possibility is that not 
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all modes of "individualiznti on" operate effectively and that our ind^x,. 
Degree of Individualization, failed Lo Lap some >crucial diffcrenc^os be- 
tween individualized programs that are effective and those that are not. 
The program characteristics measured by this index includutl >four scales: 
Individualization in Decision Making (covering decisions on topic selec- 
tion, materials selection, axyd sequencing decisions); Individualization of 
Instructional' Pace ; Use of Performance Agreements; and Utilization of 
Student Evaluation (covering the extent. ♦to which tests arc used for evalu- 
ating student needs, and whether or not such evaluation serves as a basis 
for modifying a student's instructional program). 

The amount of independence and decision making left to the student 
could also be crucial. VHiile tailorin^^ the educational process or teaching 
strategy to an individual student's needs or learning style may be very 
effective , aliovjing the student a great deal of independence in selecting 
the topics to be s'tudied, in deciding the amount of time to be spent on a 
particular topic, and in establishing the level or standard of attaimncnt 
during a school year could have a negative impact on performance (as mea- 
sured by standardized achievement testis) for all but the highly inoLivated. 
The extent to which teachers monitored, encouraged, prodtled, etc., tlielr 
students may well be crucial but this was not tapped by our data collection 
Instruments. Anecdotal experiences gained over t;]ie years of r.ssoc iat ion ' 
with* schools participating in Project LONGSTEP suggest that some students 
tend to lose momentum unless the teacher provides the kind of ihdi vid ua] 
at ten t ion that is unlikely except when a sufficient number of qualified 
assistants are available. 

This problem of student motivation was discussed byj.ipson (1974) in 
his explanation of the possible reasons why the IPI matliematics modules 
did not produce the dramatic gains tliat had been expected. To a large 
extent, his comments apply to a number of other individualized programs. 
According to Lipson, the process of each student working alone on his 
instructional module can have a very detrimental effect on motivation, in 
contrast to a conventional classroom where the group provides competition 
and a feeling of importance to, the activities. "The student's relation- 
ship to the other members of the class and to the teacher creates a sense 
of obligation for some level of performance. The rhythm of the class, 
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the rituiils of class activity — even when we gruigbJ.e — give our lives a 
pattern to follow. The progress of the class is like a tide which carries 
all the swimmers along even though some are slower and some are faster'* 
(197':, p. 60). In concluding, Lipson expresses the hor^e that new systeas 
will be developed '^which build upon the combined strengths of the classroom 
approach and the modularized and individualized approaches." 

The fact that our Degree of Individualization index failed to discrim- 
inate between successful and unsuccessful educational approaches also sug- 
gests the possibility that we may have overlooked some elements of individual- 
ization that occur in conventional classrooms. It could be that individual- 
'ization is one of the practices that distinguishes good teachers from poor 
ones, even in traditional educational environments where there is no 
"experimental" or "innovative" program claiming 'to be individualized. Per-- 
haps the. good teacher individualizes informally and so iuj? tinctively that 
he or she is not even aware of it in these terms. The good teacher does 
not treat all children as interchangeable cogs; but rather as the individuals 
they are. This kind of informal individualization would not be noted as a 
feature of an educational program in Project LONGSTEP — but it may never tiie- 
less be very effective. „ * 

This hypothesis, incidantally , if "or/ect, would extend to the elemen- 
tary school level the interpretation of some loosely comparable findings at 
the high school level, based on Project TALENT data (Shaycoft, 1967). 
Ana] yzing Proj ec t TALENT retest data, Shaycoft found evidence, as ha^ 
Project LONGSTEP, that schools differ in effectiveness. Howevdr, she failed' 
to find, as LONGSTEP has failed to find, that these differences were meaning- 
fully related to available variab].cs in any way that might have been expected 

On the other hand, it is also possible that large overall, educational 
effects were not demonstrated in Project LONGSTEP (or have not been found 
in many educational evaluation efforts) because such effects have been 
attenuated by inappropriate mauches between educational approaches and 
student needs. The undcubtedly dramatic growth in achievement demonstrated 
by a number of' students participating in Project LONGSTEP certainly sug- 
gests that some near-optimal match of student and educational approach 
may have been one of the reasons for the gains of these students. 
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In final summary, the Project: LONGSTEP findings sliouJd not be viewed 
as a sweeping criticism of the concepts of . innovation or individualization. 
However, they should serve as a reminder to educators — as well as to par- 
ents and legislators — that educate rial innovation per se will not neces- 
sarily produce dramatic effects on student acliievcment. Educational quality 
is not synonymous with ■ \nnova tion or individualization. Individualization 
of instruction may represent a valuable approach for the improvement of 
American education, but the findings here suggest that individualization as 
a program strategy should not be viewed as the final or complete answer. 
In this case, more (innovation) is not necessarily bettei (education) . 
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